Automatic dialogue design

ABSTRACT

A chatbot learns a person&#39;s related “intents” when asking for information and thereafter, in response to an initial query, which the chatbot answers, the chatbot generates a secondary dialogue, either providing the person with additional information or inquiring as to whether the person wishes to know more about a subject. The chatbot may use an external trigger such as time, event, etc. and automatically generate a query or give information to the person without any initial query from the person.

FIELD

The application relates generally to chatbots automatically designingdialogs.

BACKGROUND

Apple Siri®, Microsoft Cortana®, Google Assistant®, Amazon Alexa™ andLine Corporation Clova™ are examples of “chatbots” that audibly respondto spoken queries from people to return answers to the queries. The term“chatbot or bot” as used herein refers to a program (or the entiresystem including it) that performs dialogue communication on behalf ofhumans. A dialogue may be a combination of an utterance (such as aquery) from a person and a response from the chatbot to the utterance.The intent of the dialogue in these systems is that initiated by theperson and is based on the subject of the utterance. In this context,“intent” refers to categorizing what kind of intention the utterance ofthe person has. The chatbot responds to the person-defined intentappropriately. To this end, “entity” refers to categorizing meaningfulwords in a person's utterance after recognizing the person's intent.

SUMMARY

As understood herein, the intent of a dialogue helpfully may bepreemptively established by the chatbot instead of the person, to betterassist the person in obtaining possibly relevant, interesting, orimportant information.

Accordingly, a device includes at least one computer memory that is nota transitory signal and that comprises instructions executable by atleast one processor to receive an utterance from a person. Theinstructions are executable to access a data structure based on theutterance to retrieve a response to the utterance and to display theresponse. The instructions further are executable to, based at least inpart on the utterance, generate a secondary dialog and automaticallyplay the secondary dialogue without any further prompt from the personapart from the utterance.

In example embodiments, the response is audibly displayed, in which casethe device can include one more speakers for playing the response. Inaddition, or alternatively, the response may be visibly displayed, inwhich case the device can include one or more displays for presentingthe response.

In some implementations, the instructions may be executable to correlateutterances from the person to generate a data structure of learnedcorrelations, and to generate the secondary dialogue based at least inpart on the learned correlations. In example embodiments, theinstructions can be executable to identify at least one trigger that isnot based on an utterance by the person, and to, responsive to thetrigger, generate the secondary dialogue.

In another aspect, an apparatus includes at least one processor and atleast one computer storage with instructions executable by the processorto receive a trigger generated by a person. The instructions areexecutable to access a data structure based on the trigger to retrieve aresponse to the trigger, and to display the response. The instructionsare further executable to generate a secondary dialogue and toautomatically play the secondary dialogue without any further promptfrom the person apart from the trigger.

In another aspect, a method includes receiving a trigger generated by aperson, accessing a data structure based on the trigger to retrieve aresponse to the trigger, and displaying the response. The method alsoincludes generating a secondary dialogue and automatically playing thesecondary dialogue without any further prompt from the person apart fromthe trigger.

The details of the present application, both as to its structure andoperation, can best be understood in reference to the accompanyingdrawings, in which like reference numerals refer to like parts, and inwhich:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of social service platform consistent withpresent principles;

FIG. 1AA is a block diagram of an example system including an example inaccordance with present principles;

FIG. 1A is a schematic diagram of a vehicle (such as a driverlessvehicle) embodiment;

FIG. 1B is a schematic diagram of a mobile communication device (such asa mobile telephone) telephone embodiment;

FIG. 2 is a block diagram of an example digital assistant environment;

FIG. 3 is a flow chart of example dialogue creation logic;

FIG. 4 is a schematic diagram of a time line related to the logic ofFIG. 3;

FIG. 5 is a flow chart of example detailed logic for relatingqueries/“intents”; and

FIG. 6 is a flow chart of example logic for generating a dialoguepreemptively without a person first prompting for any information.

DETAILED DESCRIPTION

This disclosure relates generally to computer ecosystems includingaspects of consumer electronics (CE) device networks such as but notlimited to distributed computer game networks, video broadcasting,content delivery networks, virtual machines, and machine learningapplications. Note that many embodiments of the instant chatbot areenvisioned, with several, including driverless vehicles and mobiletelephones, described and shown herein.

A system herein may include server and client components, connected overa network such that data may be exchanged between the client and servercomponents. The client components may include one or more computingdevices including game consoles such as Sony Play Station® and relatedmotherboards, portable televisions (e.g. smart TVs, Internet-enabledTVs), portable computers such as laptops and tablet computers, and othermobile devices including smart phones and additional examples discussedbelow. These client devices may operate with a variety of operatingenvironments. For example, some of the client computers may employ, asexamples, Orbis or Linux operating systems, operating systems fromMicrosoft, or a Unix operating system, or operating systems produced byApple, Inc. or Google. These operating environments may be used toexecute one or more browsing programs, such as a browser made byMicrosoft or Google or Mozilla or other browser program that can accessweb sites hosted by the Internet servers discussed below. Also, anoperating environment according to present principles may be used toexecute one or more computer game programs.

Servers and/or gateways may include one or more processors executinginstructions that configure the servers to receive and transmit dataover a network such as the Internet. Or, a client and server can beconnected over a local intranet or a virtual private network. A serveror controller may be instantiated by a game console and/or one or moremotherboards thereof such as a Sony PlayStation®, a personal computer,etc.

Information may be exchanged over a network between the clients andservers. To this end and for security, servers and/or clients caninclude firewalls, load balancers, temporary storages, and proxies, andother network infrastructure for reliability and security. One or moreservers may form an apparatus that implement methods of providing asecure community such as an online social website to network members.

As used herein, instructions refer to computer-implemented steps forprocessing information in the system. Instructions can be implemented insoftware, firmware or hardware and include any type of programmed stepundertaken by components of the system.

A processor may be any conventional general-purpose single- ormulti-chip processor that can execute logic by means of various linessuch as address lines, data lines, and control lines and registers andshift registers.

Software modules described by way of the flow charts and user interfacesherein can include various sub-routines, procedures, etc. Withoutlimiting the disclosure, logic stated to be executed by a particularmodule can be redistributed to other software modules and/or combinedtogether in a single module and/or made available in a shareablelibrary.

Present principles described herein can be implemented as hardware,software, firmware, or combinations thereof; hence, illustrativecomponents, blocks, modules, circuits, and steps are set forth in termsof their functionality.

Further to what has been alluded to above, logical blocks, modules, andcircuits described below can be implemented or performed with ageneral-purpose processor, a digital signal processor (DSP), a fieldprogrammable gate array (FPGA) or other programmable logic device suchas an application specific integrated circuit (ASIC), discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. A processorcan be implemented by a controller or state machine or a combination ofcomputing devices.

The functions and methods described below, when implemented in software,can be written in an appropriate language such as but not limited toJava, C# or C++, and can be stored on or transmitted through acomputer-readable storage medium such as a random access memory (RAM),read-only memory (ROM), electrically erasable programmable read-onlymemory (EEPROM), compact disk read-only memory (CD-ROM) or other opticaldisk storage such as digital versatile disc (DVD), magnetic disk storageor other magnetic storage devices including removable thumb drives, etc.A connection may establish a computer-readable medium. Such connectionscan include, as examples, hard-wired cables including fiber optics andcoaxial wires and digital subscriber line (DSL) and twisted pair wires.Such connections may include wireless communication connectionsincluding infrared and radio.

Components included in one embodiment can be used in other embodimentsin any appropriate combination. For example, any of the variouscomponents described herein and/or depicted in the Figures may becombined, interchanged or excluded from other embodiments.

“A system having at least one of A, B, and C” (likewise “a system havingat least one of A, B, or C” and “a system having at least one of A, B,C”) includes systems that have A alone, B alone, C alone, A and Btogether, A and C together, B and C together, and/or A, B, and Ctogether, etc.

FIG. 1 shows a voice-based social service platform 150 for one or morehuman users or speakers 152 with one or more respective human friends154. A facial animation module 156 for animating one or morecomputer-displayed avatars 158 includes one or more cloud-based servers160 communicating with one or more feature analysis modules 162 tooutput lip synchronization information 164 to cause the lips of theavatar 158 to move in synchronization with a chatbot speaking, which isplayed on audio speakers. Further details of the facial animation module156 are disclosed in the present assignee's U.S. patent applicationdocket no. SCEA17103US00.

As shown in FIG. 1, the facial animation module 156 overlaps with achatbot dialogue generator module 166. The chatbot dialogue generatormodule 166 includes a local or remote (cloud) storage 168 to record andstore the voice of the speaker 152. Information from the storage 168 maybe processed by a large vocabulary continuous speech recognizer (LVCSR)module 170, which performs speech recognition on the information in thestorage 168. The LVCSR module 170 may communicate with a naturallanguage understanding unit (NLU) 172 to derive a meaning from thespeech recognized by the LVCSR 170 using, for example, machine learningprinciples.

In turn, the NLU 172 may communicate with an attribute data analysisengine 174 to derive emotion and personality indicators of the speaker152 based on the output of the NLU 172. The NLU 172 may also exchangeinformation with a dialogue generator 176 to generate additional orsecondary dialogue according to principles discussed further below. Theengines and modules in FIG. 1 may provide data to and/or access data inone or more knowledge databases 178 including third party databases(such as hotel and restaurant review services). The engines and modulesin FIG. 1 may use processing capability in a network-based big dataprocessing server center 180.

FIG. 1AA shows a system 10 to illustrate example hardware and softwarecomponents used by the various computer-based components herein shown,and which may include one or more of the example devices mentioned aboveand described further below in accordance with present principles. Thefirst of the example devices included in the system 10 is a consumerelectronics (CE) device such as an audio video device (AVD) 12 such asbut not limited to an Internet-enabled TV with a TV tuner (equivalently,set top box controlling a TV). However, the AVD 12 alternatively may bean appliance or household item, e.g. computerized Internet enabledrefrigerator, washer, or dryer. The AVD 12 alternatively may also be acomputerized Internet enabled (“smart”) telephone, a tablet computer, anotebook computer, a wearable computerized device such as e.g.computerized Internet-enabled watch, a computerized Internet-enabledbracelet, other computerized Internet-enabled devices, a computerizedInternet-enabled music player, computerized Internet-enabled headphones, a computerized Internet-enabled implantable device such as animplantable skin device, etc. Regardless, it is to be understood thatthe AVD 12 is configured to undertake present principles (e.g.communicate with other CE devices to undertake present principles,execute the logic described herein, and perform any other functionsand/or operations described herein).

Accordingly, to undertake such principles the AVD 12 can be establishedby some or all of the components shown in FIG. 1AA. For example, the AVD12 can include one or more displays 14 that may be implemented by a highdefinition or ultra-high definition “4K” or higher flat screen and thatmay be touch-enabled for receiving user input signals via touches on thedisplay. The AVD 12 may include one or more speakers 16 for outputtingaudio in accordance with present principles, and at least one additionalinput device 18 such as e.g. an audio receiver/microphone for e.g.entering audible commands to the AVD 12 to control the AVD 12. Theexample AVD 12 may also include one or more network interfaces 20 forcommunication over at least one network 22 such as the Internet, an WAN,an LAN, etc. under control of one or more processors 24. Thus, theinterface 20 may be, without limitation, a Wi-Fi transceiver, which isan example of a wireless computer network interface, such as but notlimited to a mesh network transceiver. It is to be understood that theprocessor 24 controls the AVD 12 to undertake present principles,including the other elements of the AVD 12 described herein such as e.g.controlling the display 14 to present images thereon and receiving inputtherefrom. Furthermore, note the network interface 20 may be, e.g., awired or wireless modem or router, or other appropriate interface suchas, e.g., a wireless telephony transceiver, or Wi-Fi transceiver asmentioned above, etc.

In addition to the foregoing, the AVD 12 may also include one or moreinput ports 26 such as, e.g., a high definition multimedia interface(HDMI) port or a USB port to physically connect (e.g. using a wiredconnection) to another CE device and/or a headphone port to connectheadphones to the AVD 12 for presentation of audio from the AVD 12 to auser through the headphones. For example, the input port 26 may beconnected via wire or wirelessly to a cable or satellite source 26 a ofaudio video content. Thus, the source 26 a may be, e.g., a separate orintegrated set top box, or a satellite receiver. Or, the source 26 a maybe a game console or disk player containing content that might beregarded by a user as a favorite for channel assignation purposesdescribed further below. The source 26 a when implemented as a gameconsole may include some or all of the components described below inrelation to the CE device 44.

The AVD 12 may further include one or more computer memories 28 such asdisk-based or solid-state storage that are not transitory signals, insome cases embodied in the chassis of the AVD as standalone devices oras a personal video recording device (PVR) or video disk player eitherinternal or external to the chassis of the AVD for playing back AVprograms or as removable memory media. Also in some embodiments, the AVD12 can include a position or location receiver such as but not limitedto a cellphone receiver, GPS receiver and/or altimeter 30 that isconfigured to e.g. receive geographic position information from at leastone satellite or cellphone tower and provide the information to theprocessor 24 and/or determine an altitude at which the AVD 12 isdisposed in conjunction with the processor 24. However, it is to beunderstood that that another suitable position receiver other than acellphone receiver, GPS receiver and/or altimeter may be used inaccordance with present principles to e.g. determine the location of theAVD 12 in e.g. all three dimensions.

Continuing the description of the AVD 12, in some embodiments the AVD 12may include one or more cameras 32 that may be, e.g., a thermal imagingcamera, a digital camera such as a webcam, and/or a camera integratedinto the AVD 12 and controllable by the processor 24 to gatherpictures/images and/or video in accordance with present principles. Alsoincluded on the AVD 12 may be a Bluetooth transceiver 34 and other NearField Communication (NFC) element 36 for communication with otherdevices using Bluetooth and/or NFC technology, respectively. An exampleNFC element can be a radio frequency identification (RFID) element.

Further still, the AVD 12 may include one or more auxiliary sensors 37(e.g., a motion sensor such as an accelerometer, gyroscope, cyclometer,or a magnetic sensor, an infrared (IR) sensor, an optical sensor, aspeed and/or cadence sensor, a gesture sensor (e.g. for sensing gesturecommand), etc.) providing input to the processor 24. The AVD 12 mayinclude an over-the-air TV broadcast port 38 for receiving OTH TVbroadcasts providing input to the processor 24. In addition to theforegoing, it is noted that the AVD 12 may also include an infrared (IR)transmitter and/or IR receiver and/or IR transceiver 42 such as an IRdata association (IRDA) device. A battery (not shown) may be providedfor powering the AVD 12.

Still referring to FIG. 1AA, in addition to the AVD 12, the system 10may include one or more other CE device types. In one example, a firstCE device 44 may be used to control the display via commands sentthrough the below-described server while a second CE device 46 mayinclude similar components as the first CE device 44 and hence will notbe discussed in detail. In the example shown, only two CE devices 44, 46are shown, it being understood that fewer or greater devices may beused. As alluded to above, the CE device 44/46 and/or the source 26 amay be implemented by a game console. Or, one or more of the CE devices44/46 may be implemented by devices sold under the trademarks GoogleChromecast, Roku, Amazon FireTV. A CE device may be established by adigital assistant, an example of which is shown and described furtherbelow.

In the example shown, to illustrate present principles all three devices12, 44, 46 are assumed to be members of an entertainment network in,e.g., a home, or at least to be present in proximity to each other in alocation such as a house. However, for present principles are notlimited to a particular location, illustrated by dashed lines 48, unlessexplicitly claimed otherwise.

The example non-limiting first CE device 44 may be established by anyone of the above-mentioned devices, for example, a digital assistant, aportable wireless laptop computer or notebook computer or gamecontroller (also referred to as “console”), and accordingly may have oneor more of the components described below. The second CE device 46without limitation may be established by a video disk player such as aBlu-ray player, a game console, and the like. The first CE device 44 maybe a remote control (RC) for, e.g., issuing AV play and pause commandsto the AVD 12, or it may be a more sophisticated device such as a tabletcomputer, a game controller communicating via wired or wireless linkwith a game console implemented by the second CE device 46 andcontrolling video game presentation on the AVD 12, a personal computer,a wireless telephone, etc.

Accordingly, the first CE device 44 may include one or more displays 50that may be touch-enabled for receiving user input signals via toucheson the display. The first CE device 44 may include one or more speakers52 for outputting audio in accordance with present principles, and atleast one additional input device 54 such as e.g. an audioreceiver/microphone for e.g. entering audible commands to the first CEdevice 44 to control the device 44. The example first CE device 44 mayalso include one or more network interfaces 56 for communication overthe network 22 under control of one or more CE device processors 58.Thus, the interface 56 may be, without limitation, a Wi-Fi transceiver,which is an example of a wireless computer network interface, includingmesh network interfaces. It is to be understood that the processor 58controls the first CE device 44 to undertake present principles,including the other elements of the first CE device 44 described hereinsuch as e.g. controlling the display 50 to present images thereon andreceiving input therefrom. Furthermore, note the network interface 56may be, e.g., a wired or wireless modem or router, or other appropriateinterface such as, e.g., a wireless telephony transceiver, or Wi-Fitransceiver as mentioned above, etc.

In addition to the foregoing, the first CE device 44 may also includeone or more input ports 60 such as, e.g., a HDMI port or a USB port tophysically connect (e.g. using a wired connection) to another CE deviceand/or a headphone port to connect headphones to the first CE device 44for presentation of audio from the first CE device 44 to a user throughthe headphones. The first CE device 44 may further include one or moretangible computer readable storage medium 62 such as disk-based orsolid-state storage. Also in some embodiments, the first CE device 44can include a position or location receiver such as but not limited to acellphone and/or GPS receiver and/or altimeter 64 that is configured toe.g. receive geographic position information from at least one satelliteand/or cell tower, using triangulation, and provide the information tothe CE device processor 58 and/or determine an altitude at which thefirst CE device 44 is disposed in conjunction with the CE deviceprocessor 58. However, it is to be understood that that another suitableposition receiver other than a cellphone and/or GPS receiver and/oraltimeter may be used in accordance with present principles to e.g.determine the location of the first CE device 44 in e.g. all threedimensions.

Continuing the description of the first CE device 44, in someembodiments the first CE device 44 may include one or more cameras 66that may be, e.g., a thermal imaging camera, a digital camera such as awebcam, and/or a camera integrated into the first CE device 44 andcontrollable by the CE device processor 58 to gather pictures/imagesand/or video in accordance with present principles. Also included on thefirst CE device 44 may be a Bluetooth transceiver 68 and other NearField Communication (NFC) element 70 for communication with otherdevices using Bluetooth and/or NFC technology, respectively. An exampleNFC element can be a radio frequency identification (RFID) element.

Further still, the first CE device 44 may include one or more auxiliarysensors 72 (e.g., a motion sensor such as an accelerometer, gyroscope,cyclometer, or a magnetic sensor, an infrared (IR) sensor, an opticalsensor, a speed and/or cadence sensor, a gesture sensor (e.g. forsensing gesture command), etc.) providing input to the CE deviceprocessor 58. The first CE device 44 may include still other sensorssuch as e.g. one or more climate sensors 74 (e.g. barometers, humiditysensors, wind sensors, light sensors, temperature sensors, etc.) and/orone or more biometric sensors 76 providing input to the CE deviceprocessor 58. In addition to the foregoing, it is noted that in someembodiments the first CE device 44 may also include an infrared (IR)transmitter and/or IR receiver and/or IR transceiver 78 such as an IRdata association (IRDA) device. A battery (not shown) may be providedfor powering the first CE device 44. The CE device 44 may communicatewith the AVD 12 through any of the above-described communication modesand related components.

The second CE device 46 may include some or all of the components shownfor the CE device 44. Either one or both CE devices may be powered byone or more batteries.

Now in reference to the afore-mentioned at least one server 80, itincludes at least one server processor 82, at least one tangiblecomputer readable storage medium 84 such as disk-based or solid-statestorage. In an implementation, the medium 84 includes one or more solidstate storage drives (SSDs). The server also includes at least onenetwork interface 86 that allows for communication with the otherdevices of FIG. 1AA over the network 22, and indeed may facilitatecommunication between servers and client devices in accordance withpresent principles. Note that the network interface 86 may be, e.g., awired or wireless modem or router, Wi-Fi transceiver, or otherappropriate interface such as, e.g., a wireless telephony transceiver.The network interface 86 may be a remote direct memory access (RDMA)interface that directly connects the medium 84 to a network such as aso-called “fabric” without passing through the server processor 82. Thenetwork may include an Ethernet network and/or Fibre Channel networkand/or InfiniBand network. Typically, the server 80 includes multipleprocessors in multiple computers referred to as “blades” that may bearranged in a physical server “stack”.

Accordingly, in some embodiments the server 80 may be an Internet serveror an entire “server farm”, and may include and perform “cloud”functions such that the devices of the system 10 may access a “cloud”environment via the server 80 in example embodiments for, e.g., networkgaming applications, digital assistant applications, etc. Or, the server80 may be implemented by one or more game consoles or other computers inthe same room as the other devices shown in FIG. 1AA or nearby.

The methods herein may be implemented as software instructions executedby a processor, suitably configured application specific integratedcircuits (ASIC) or field programmable gate array (FPGA) modules, or anyother convenient manner as would be appreciated by those skilled inthose art. Where employed, the software instructions may be embodied ina non-transitory device such as a CD ROM or Flash drive. The softwarecode instructions may alternatively be embodied in a transitoryarrangement such as a radio or optical signal, or via a download overthe internet.

FIG. 1A shows a specific non-limiting example in which a system 100includes a vehicle 102 such as a driverless vehicle in which a chatbotapplication consistent with present principles has been downloaded fromthe cloud, such as a server 80, onto one or more computer memories 104,which may be implemented by any of the computer storage devicesdescribed herein. The chatbot application can be executed by one or moreprocessors 106 to output information as further disclosed below on oneor more output devices, including a visual display 108 such as a flatpanel display, a tactile signal generator 110 such as a buzzer or otherdevice that generates tactile signals, and one or more audio speakers112. The processor 106 may receive input from one or more sensors 114such as microphones, cameras, biometric sensors. The processor 106 maycommunicate with a network such as the Internet using one or more wiredor more typically wireless network interfaces 116 such as but notlimited to Wi-Fi.

FIG. 1B shows another specific non-limiting example in which a system100A includes a mobile communication device (MCD) 102A such as a mobiletelephone in which a chatbot application consistent with presentprinciples has been downloaded from the cloud, such as a server 80, ontoone or more computer memories 104A, which may be implemented by any ofthe computer storage devices described herein. The chatbot applicationcan be executed by one or more processors 106A to output information asfurther disclosed below on one or more output devices, including avisual display 108A such as a flat panel display, a tactile signalgenerator 110A such as a buzzer or other device that generates tactilesignals, and one or more audio speakers 112A. The processor 106A mayreceive input from one or more sensors 114A such as microphones,cameras, biometric sensors. The processor 106A may communicate with anetwork such as the Internet using one or more wired or more typicallywireless network interfaces 116A such as but not limited to Wi-Fi. TheMCD may also include one or more wireless telephony transceivers 118Asuch as but not limited to code division multiple access (CDMA)transceivers, global system for mobile communication (GSM) transceivers,etc.

FIG. 2 illustrates an example application of a CE device 44 implementedby a digital assistant 200 communicating, via a network interface 202such as a Wi-Fi or other appropriate wired or wireless interface withthe Internet 204 and thence with one or more servers 80 to exchangeinformation therewith. A person 206 can speak into a microphone 208 ofthe digital assistant 200 and the person's voice is digitized foranalysis using speech recognition by a processor 210 accessinginstructions on a computer memory or storage 212 such as disk-based orsolid-state storage. The digital assistant responds to queries from theperson 206 by accessing data on the server 80 and/or storage 212 andconverting query results to audible signals that are played on one ormore speakers 214 and/or presented on one or more visual displays 216.

FIG. 3 illustrates example overall logic consistent with presentprinciples. At block 300, queries or other voice input from the speaker152 shown in FIG. 1 may be received over time. Concurrent emotions ofthe speakers, including voice tones, may be received and correlated withthe voice input, digitized, and stored locally or remotely on a cloudserver.

To determine emotion, images of the speaker may be taken, and facialfeature recognition executed on them to correlate the facial features toemotions such as happiness, anger, etc. Other biometric sensors inaddition to image recognition of facial features in digital images maybe used to determine concurrent emotion of the speaker. U.S. Pat. Nos.9,798,920, 9774,690, 9,298,977, and 9,031,293, all incorporated hereinby reference and all owned by Sony Corp. or a subsidiary thereofdescribe various techniques for determining the emotion of a person. Inthe vehicle example above, driving actions may be correlated to emotion.For instance, a hard-braking action may indicate fear or anger, while asoft braking action may be correlated to a relaxed state or happiness.Similarly, vigorous rotation of the steering wheel or forcefulapplication of the accelerator pedal may indicate tension while slowerrotation of the steering wheel and gentle application of the acceleratormay be correlated to a relaxed state or happiness. Gaze tracking andrapidity of movements such as in gently or forcefully opening the doorand closing the vehicle door may be correlated to tension/relaxation,respectively. Wearable biometric-sending devices can be used to detectsuch things as heart rate and perspiration and correlated to emotions ofvehicle occupants. For the vehicle implementation of FIG. 1A, theprocessing device processing voice and emotion signals may be a separateprocessor in, e.g., a user's mobile telephone, or an onboard computer,or by a processor receiving the information from the vehicle wirelesslyusing any of the transceivers described herein and executing emotionrecognition in the cloud.

Moving to block 302, related queries are learned over time. Moreparticularly, machine learning may be employed to learn queries the usermay often link together, with date and time characteristics of therelationships being correlated with the relationships. As an example, ifa speaker requests to know the time and then habitually follows up thatquery with one about the weather, the weather query can be correlatedwith the time query along with times of day and days of the week thequeries typically are related by the user.

Proceeding to block 304, the system implementing the present chatbotresponds to a received query and at block 306 uses the query to access adata structure of related queries to identify a related query. When sucha related query is identified (at first, default relationships may beused while machine learning is implemented on user queries over time), asecondary dialogue based on the related query is created and played atblock 308. For example, in the example above, if the user requests toknow the time of day, at block 304 the chatbot may respond with the timeof day and then at block 308 the chatbot may ask “would you like to knowthe weather?” or may preemptively inform the user of the current weatherconditions and/or forecast. The secondary dialogue may be based in parton the sensed emotion of the user. For example, if the user is sensed asbeing tense, a secondary dialogue may be delayed or changed to somethingrelatively anodyne or soothing, such as “it's a beautiful day” in anattempt to calm the user, whereas if the user is sensed as beingrelaxed, secondary dialogs may be immediately presented even if theyrequire additional attention or response from the user.

While chatbots typically produce audio feedback to the user usingspeakers 214 as shown in FIG. 1 and in some cases, can present visibleresponses in displays 216, haptic feedback may also or alternatively bepresented as described above in relation to FIGS. 1A and 1B. That is, asecondary dialogue presented at block 308 may be a haptic buzz or otherhaptic alert that additional information is available if the user wouldask, for example. Haptic feedback may be particularly appropriate forthe vehicle application of FIG. 1A, and also feedback on an onboarddisplay, through speakers, dashboard displays, car lighting (e.g.,changing color to indicate a response or warning), etc. As a safetyfeature, other vehicle drivers may be signaled using the vehicle'sexternal lamps that the vehicle driver is having a health crisis. Otherinputs/factors that a chatbot may consider include temperature, currentweather, time of day, day of week, mode of transportation. Aplugin/wireless device may be provided for the vehicle to determine theuser's driving patterns and determine the relative level of safety ofthe driving as indicated by speed sensors, force sensors on pedals,rotation sensors on the steering wheel, etc.

In some implementations such as the vehicle implementation of FIG. 1A,instead of a voice input triggering the chatbot and/or instead ofsecondary dialogue being presented in response to a user-generatedinitial dialogue, other things could trigger a chatbot, such as a sensorindication that a vehicle door has been opened, a wireless signal froman authorized vehicle key fob has been received (to energize a lampinside the vehicle, for instance), a seat has been sat in, oridentifying that a Wi-Fi or Bluetooth connection has been establishedwith the vehicle. In some implementations, a wakeup word per se need notbe used to trigger a chatbot, but rather the chatbot module may listento conversation and based on the context as output by, e.g., the NCU 172of FIG. 1, generate a secondary dialogue and play it at block 308 ofFIG. 3.

The features of FIGS. 1A and 1B may be combined. For example, a mobiletelephone can be used with a vehicle or be carried by the user, and usermovement within the vehicle (train, car or airplane) as sensed by, e.g.,a GPS receiver or accelerometer in the mobile device may be identifiedto determine a speed of movement of the user, with slower speeds beingcorrelated to a calm emotion and faster speeds to hurried emotions. Thechatbot application accordingly can sense if the user is in a public orprivate place, so if in a public place for example, to preserve privacy,audio output may be sent to wireless headphones instead of theloudspeakers of the device.

Other secondary dialogs may include presenting a user interface (UI) ofmobile telephone traffic accidents on the route the user normally takesto work in the morning. A photograph of a destination may be presentedas well as a logo related to the destination. A warning implemented by aflashing light on the mobile device may be used as an advisory to leaveearlier based on current traffic conditions.

Dialogue generation may start from knowing it rained, so the chatbot appcan determine to output a wakeup signal such as a buzz or audible toneearlier than the user may have set the alarm time to be. The secondarydialogue may preemptively inquire on speakers whether the user wants tosend a co-worker notice that the user likely will be late based on theinclement weather.

Emotion as sensed as described above and/or by a drive recorder orelectronic on-board recorder in the vehicle may be used to establish atone of voice or other audible characteristic of how the chatbotresponds to queries. If the user speaks slowly, for instance, thechatbot may speak a response at a relatively slow speed to mirror theuser, while if the user speaks rapidly, the speech is output by thechatbot at a relatively fast rate. As indicated above, if the user'semotions indicate she is rushed or nervous the chatbot may respond witha slower rate of speech and soft tone to calm the user.

The present chatbot module may be implemented by a CE device embodied asa virtual reality (VR) headset or kiosk in a public place. For a kioskimplementation, if the user speaks her native tongue the chatbot appresponds in the user's native tongue even if in a country in which theprincipal language is not the user's native tongue. If the user wishesfor the chatbot app to respond in the language of the country in whichthe principal language is not the user's native tongue, the chatbot mayrespond to user input to this end by presenting a response to a queryreceived in the user's native tongue in the (different) language of thelocal country, e.g., by presenting a text response on a display of amobile telephone to show to a native of the country, who may thus bebetter able to help.

FIG. 4 schematically illustrates a timeline 400 of an exampleuser-chatbot interaction with user input shown above the line 400 andchatbot actions shown below it. A user may enter a wakeup signal 402 (byspeaking a wakeup word, by undertaking a predetermined wakeup action,etc.) which enables chatbot response at 404. At 406 the chatbot checksif one or more necessary conditions for interaction are met, e.g.,whether a network connection exists, and if a necessary condition is notmet an error message 408 is returned and the process may end at 410 witha message such as “check network connection”.

If necessary, conditions are met the chatbot waits for a query or otherutterance at 412. If no follow-up utterance to the wakeup is receivedwithin a timeout period, a warning may be audibly or visibly ortactilely presented on an output device such as any of those describedherein at 414. The warning may be a prompt such as “I can't hear you” toalert the user to speak up.

On the other hand, if the user enters, as by speaking, an utterance 416within the timeout period, the intent of the utterance is recognized bythe chatbot app at 418 according to principles discussed above. If nointent is recognized, i.e., the chatbot has no information to give tothe user in response to the utterance, a warning message 420 may bepresented.

If the intent is recognized, however, one or more entities (e.g.,information sources) may be extracted at 422 and queried at 424. Forexample, a clock entity may be accessed and queried for the time.

Also, at 426 in accordance with principles above additional informationincluding information from the entities 422 may be correlated to theutterance 416 and presented as responses to the query-utterance 416, byresponding to the utterance at 428 and/or by posing a secondary questionrelated to the utterance 416 and/or response 428 at 430. As describedabove, this may be done using machine learning of the user's pastbehaviors including related utterances to answer the utterance 416 andpropose a next action.

Intent may then be confirmed starting at state 432 by awaiting a furtherutterance from the user. If not received within a timeout period, awarning message may be generated at 434. However, if within the timeoutperiod a secondary user utterance 436 is received, such as a response tothe question 428 or next action proposed at 430, the intent of thesecondary utterance 436 is recognized (or not) at state 438. If nointent is recognized, i.e., the chatbot has no information to give tothe user in response to the utterance or cannot understand theutterance, a warning message 440 may be presented such as “please sayagain”.

If the intent is recognized, however, one or more entities may beextracted at 442 and queried at 444. Action based on information fromthe entities is then used to take the action required by the secondaryutterance 436 at state 446. The logic may then stop at 448.

FIGS. 5 and 6 illustrate additional logic consistent with presentprinciples. As mentioned above, while logic herein may be shown in flowchart format for illustration purposes in a deterministic and step bystep style, in implementation the logic may include other styles such asstate logic and/or a Neural Network (NN) which receives theabove-discussed input parameters as input, rendering a decision as anoutput, with the NLU functionality performed within the NN.

At block 500 in FIG. 5, related queries/utterances may be linkedtogether in a data structure and also categorized by time of day.Likewise, related queries/utterances may be categorized at block 502 byday of the week the user's actions typically relate them together.Moreover, at block 504 related queries/utterances can be correlated toentries in the user's e-calendar as may be stored on the user's mobiletelephone.

Using, for example, the NLU in FIG. 1, related queries/utterances may becategorized by subject matter at block 506 and ranked at block 508according to which related queries/utterances are most popular withother users.

Then, at FIG. 6 a trigger may be received such as any of the non-verbaltriggers mentioned above. However, the trigger may also be verbalconsistent with present principles.

Proceeding to block 602, a data structure of related queries/utterancesestablished according to FIG. 5 in one example may be accessed todetermine related “intents” of the user. At block 604 a dialogue may becreated and played based on the related intents on a speaker and/orvisual display and/or haptic generator.

It will be appreciated that whilst present principals have beendescribed with reference to some example embodiments, these are notintended to be limiting, and that various alternative arrangements maybe used to implement the subject matter claimed herein.

What is claimed is:
 1. A device comprising: at least one processorconfigured with instructions to: receive an utterance; identify anemotion based at least in part on the utterance; identify a tone ofvoice, a rate of speech, or both a tone of voice and a rate of speechbased at least in part on the emotion; and display a response to theutterance using the tone of voice, the rate of speech, or both the toneof voice and rate of speech.
 2. The device of claim 1, comprising atleast one speaker for playing the response.
 3. The device of claim 1,wherein the response is visibly displayed.
 4. The device of claim 3,comprising at least one display for presenting the response.
 5. Thedevice of claim 1, wherein the response is audibly displayed.
 6. Thedevice of claim 1, wherein the instructions are executable to: correlateutterances from the person to generate a data structure of learnedcorrelations; and generate a secondary dialogue to display afterdisplaying the response based at least in part on the learnedcorrelations.
 7. The device of claim 1, wherein the instructions areexecutable to: identify at least one trigger that is not based on anutterance by the person; and responsive to the trigger, generate asecondary dialogue to display after displaying the response.
 8. Anapparatus, comprising: at least one processor configured; withinstructions executable by the at least one processor to: receive anutterance; identify an emotion based at least in part on the utterance;identify an emotion based at least in part on the utterance; based atleast in part on the emotion, identify a tone of voice, a rate ofspeech, or both a tone of voice and a rate of speech; and display theresponse to the utterance using the tone of voice, the rate of speech,or both the tone of voice and rate of speech.
 9. The apparatus of claim8, wherein a trigger associated with the utterance comprises an actionrelated to a vehicle.
 10. The apparatus of claim 8, wherein theinstructions are executable to: generate a secondary dialogue based atleast in part on the emotion; and automatically play the secondarydialogue without any further prompt from the person apart from theutterance.
 11. The apparatus of claim 10, comprising at least onespeaker for playing the response.
 12. The apparatus of claim 8, whereinthe response is visibly displayed.
 13. The apparatus of claim 12,comprising at least one display for presenting the response.
 14. Theapparatus of claim 8, wherein the instructions are executable to:correlate utterances from the person to generate a data structure oflearned correlations; and generate the secondary dialogue based at leastin part on the learned correlations.
 15. The apparatus of claim 8,wherein the instructions are executable to: identify at least onetrigger that is not based on an utterance by the person; and responsiveto the trigger, generate the secondary dialogue.
 16. A method,comprising: receiving an utterance from a person; identifying an emotionof the person based at least in part on the utterance; identifying atone of voice, a rate of speech, or both a tone of voice and a rate ofspeech based at least in part on the emotion; and using at least oneaudio speaker, displaying a response to the utterance using the tone ofvoice, the rate of speech, or both the tone of voice and rate of speech.17. The method of claim 16, wherein the trigger comprises indicationthat a vehicle door has been opened.
 18. The method of claim 16, whereinthe trigger comprises indication that a wireless signal from anauthorized vehicle key fob has been received.
 19. The method of claim16, wherein the trigger comprises indication that a wireless connectionhas been established with a vehicle.